Architectures and methods for creating and representing time-dependent imagery

ABSTRACT

The present invention pertains to geographical image processing of time-dependent imagery. Various assets acquired at different times are stored and processing according to acquisition date in order to generate one or more image tiles for a geographical region of interest. The different image tiles are sorted based on asset acquisition date. Multiple image tiles for the same region of interest may be available. In response to a user request for imagery as of a certain date, one or more image tiles associated with assets from prior to that date are used to generate a time-based geographical image for the user.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.13/939,628, filed on Jul. 11, 2013, which is a continuation of U.S.application Ser. No. 13/619,183, filed on Sep. 14, 2012, which is acontinuation of U.S. application Ser. No. 13/285,250, filed on Oct. 31,2011 and issued as U.S. Pat. No. 8,295,550 on Oct. 23, 2012, which is acontinuation of U.S. application Ser. No. 12/231,290, filed on Aug. 28,2008 and issued as U.S. Pat. No. 8,077,918 on Dec. 13, 2011, the entiredisclosures of which are incorporated herein by reference.

BACKGROUND

1. Field of the Invention

The present invention relates generally to image mapping systems andmethods. More particularly, the present invention relates to blendingimagery in a geographical mapping environment.

2. Description of Related Art

Imagery taken by satellites, planes and other sources has long been usedto provide visual information about the earth. Recently, dramaticimprovements in computer processing power and broadband streamingtechnology have lead to the development of interactive systems fornavigating imagery (e.g., map imagery). Some interactive map navigationsystems provide a user interface (“UI”) with navigation controls fordynamically navigating cities, neighborhoods and other terrain in threedimensions. The navigation controls enable users to tilt, pan, rotate,zoom and activate terrain and buildings for different perspectives at apoint of interest. An example of an interactive 3D map system fornavigating Earth imagery is Google Earth™ developed by Google Inc.(Mountain View, Calif.).

The production imagery used by interactive map navigation systems istypically derived by processing large pieces of geo-located imagery or“assets.” Such assets can be taken from a single pass of a satellite,airplane or other airborne platform, or can be stitched together frommultiple aerial images. Once the assets are processed, they can be movedto datacenters where it can be distributed to client devices.

Different assets may have different resolutions and/or may be capturedat different points in time. Large quantities of new assets arecollected on an ongoing basis. Unfortunately, processing such assets canbe a complex and time consuming process. Furthermore, image storage anddatabase maintenance may be problematic due to an ever-expanding amountof assets. Aspects of the present invention address these and otherproblems.

SUMMARY OF THE INVENTION

Architectures and methods that process and distribute time-dependentassets to users are provided herein.

In accordance with one embodiment of the present invention, a methodcomprises providing an identification of available points in time forwhich images are available for a geographical location, theidentification being provided by a processor of a computer; receiving arequest for an image associated with the geographical location for oneof the available points in time at the computer; and in response to therequest, the computer providing the image associated with the requestedgeographical location. Portions of the provided image comprise differentimages of the geographical location captured at different points intime. The different images are selected from a plurality of imagescomprising images captured before and after the requested point in time.Furthermore, the different images included in the provided image werecaptured prior to the requested point in time.

In one alternative, the method further comprises deriving the pluralityof different images from assets obtained from an imagery source; andidentifying the point in time associated with each of the plurality ofdifferent images, each point in time corresponding to an acquisitiondate of a respective asset.

In this case, the method may further comprise generating a blended imagefor the requested point in time. Here, the blended image includes atleast one secondary image from a point in time earlier than therequested point in time and a primary image from the requested point intime. In this case the primary image overlies the secondary image.

In another case, generating the blended image for the requested point intime includes generating multiple blended images each having a differentlevel of detail. In this case, the request for an image associated withthe geographical location may further include a request for a minimumlevel of detail and wherein the provided image is one of the blendedimages having the minimum level of detail.

In accordance with another embodiment of the present invention, a methodof processing geographical imagery comprises obtaining imagery from animagery source; identifying an acquisition date for at least some of theimagery obtained from the imagery source; blending overlapping pieces ofimagery with a processor of a computer based on respective acquisitiondates to form one or more image tiles associated with each acquisitiondate; storing the one or more tiles in memory associated with thecomputer; and providing at least one of the image tiles having aselected acquisition date from the memory to a user upon request for animage associated with a geographical location for the selectedacquisition date.

In one alternative, blending the overlapping pieces of imagery based onthe respective acquisition dates forms multiple tiles for a givenacquisition date. Here, the overlapping pieces of imagery are layeredchronologically by acquisition date. In one example, the piece ofimagery with the most recent acquisition date overlies the other piecesof imagery with earlier acquisition dates. In another example, at leastsome of the multiple image tiles incorporate the same overlapping piecesof imagery at different levels of detail.

In accordance with a further embodiment of the present invention, aprocessing system for processing geographical imagery comprises at leastone processor and memory for storing data. The memory is electricallycoupled to the at least one processor. The at least one processor isoperable to obtain imagery from an imagery source, to identify anacquisition date for at least some of the imagery obtained from theimagery source, to blend overlapping pieces of imagery based onrespective acquisition dates to form one or more image tiles associatedwith each acquisition date, to store the one or more image tiles in thememory, and to provide at least one image tile having a selectedacquisition date from the memory to a user upon request for an imageassociated with a geographical location for the selected acquisitiondate.

In an example, the at least one processor is operable to blend theoverlapping pieces of imagery based on the respective acquisition datesto form multiple image tiles for each acquisition date. Here, theoverlapping pieces of imagery are layered chronologically by acquisitiondate. In one alternative, the piece of imagery with the most recentacquisition date overlies the other pieces of imagery with earlieracquisition dates. In another alternative, at least some of the multipleimage tiles incorporate the same overlapping pieces of imagery atdifferent levels of detail.

In accordance with yet another embodiment of the present invention, acomputer-readable medium having instructions stored thereon is provided.The instructions, when executed by a processor, cause the processor toperform the operations of obtaining imagery from an imagery source;identifying an acquisition date for at least some of the imageryobtained from the imagery source; blending overlapping pieces of imagerybased on respective acquisition dates to form one or more image tilesassociated with each acquisition date; storing the one or more tiles inmemory; and providing at least one image tile having a selectedacquisition date from the memory to a user upon request for an imageassociated with a geographical location for the selected acquisitiondate.

In accordance with another embodiment of the present invention, a methodof processing imagery comprises blending a plurality of tilesrepresentative of a geographical location using a processor, at leastsome of the tiles being time-based tiles associated with imagery of thegeographical location from different acquisition dates; preparing atiles table for organizing the plurality of tiles, the tiles table beingindexed by location and level of detail for each of the plurality oftiles; creating fingerprints having tile creation information for eachof the plurality of tiles; the processor generating a plurality ofpackfiles, each packfile being associated with at least one of theplurality of tiles; and distributing the plurality of packfiles to atleast one datacenter; wherein the tiles table further indexes thetime-based tiles by acquisition date.

In one example, blending the time-based tiles includes blendingoverlapping pieces of imagery based on respective acquisition dates. Inanother example, a given packfile contains at least one of a given tile,an indication that the given tile is shared with a database, or adeletion marker indicating that a tile entry is to be deleted.

In an alternative, the method further comprises updating a respectiveone of the fingerprints after tile information associated with therespective fingerprint has been incorporated into a packfile. In thiscase, the method may also comprise updating the tiles table based on theupdated fingerprint.

In yet another alternative, the method further comprises indexing thedistributed packfiles; updating an image data table based upon thedistributed packfiles; and updating a quadtree packet table based uponthe indexed packfiles; wherein each distributed packfile havingtime-based information therein is further indexed based upon thetime-based information. In one example, the method further comprisesstoring the image data table and the quadtree packet table in a databaseof an image server.

In accordance with another embodiment of the present invention, a systemfor managing imagery is provided. The system includes means for indexingtime-based packfiles and non-time-based packfiles. Each packfilecontains at least one of an image tile, an indication that the at leastone image tile is shared with a database, or a deletion markerindicating that an image tile entry in the database is to be deleted.The system also includes means for updating at least one image datatable based upon the time-based and non-time-based packfiles. The atleast one image data table includes image data for generated imagetiles. The system also includes means for updating at least one quadtreepacket table based upon the indexed packfiles and means for distributingquadtree packets of the quadtree packet table and image data of the atleast one image data table to a client device upon request.

In one example, the means for indexing the packfiles indexes thetime-based packfiles in a time-based index table and indexes thenon-time-based packfiles in a non-time-based index table. Each indextable includes a version indicator and a location indicator while thetime-based-index table further includes a date indicator.

In another example, the at least one quadtree packet table includes afirst quadtree packet table for managing time-based quadtree packets anda second quadtree packet table for managing non-time-based quadtreepackets.

In an alternative, the system further comprises means for creatingfingerprints. Each of the fingerprints has tile creation information fora respective one of the image tiles.

In another alternative, the system further comprising means forprocessing selected image tiles based upon the fingerprints. In thiscase, the means for processing may be operable to blend an existingimage tile with a new image tile having time information associatedtherewith. Here, whether blending is performed is based on a comparisonof the fingerprint of the existing image tile with the fingerprint ofthe new time-based image tile.

In accordance with yet another embodiment of the present invention, animage processing method comprises a processor requesting quadtreepackets for a geographical location, the quadtree packets containingtile information for the geographical location, at least some of thequadtree packets including date information for respective tilesassociated therewith; the processor requesting one or more tiles havinga specified date for the geographical location; and the processorpresenting at least one of the requested tiles on a display, whereinportions of the at least one of the requested tiles comprise differentimages of the geographical location captured at different points intime, the different images being selected from a plurality of imagescomprising images captured before and after the specified date, andwherein the different images included in the presented tile werecaptured prior to the specified date.

In one alternative, the portions are blended chronologically. In anotheralternative, quadtree packets further include level of detailinformation for the respective tiles. Here, requesting the one or moretiles further includes identifying a specific level of detail to bepresented on the display.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flow diagram in accordance with aspects of the presentinvention.

FIGS. 2A-G illustrate examples of assets and tiles for a region ofinterest.

FIGS. 3A-B illustrate a computer system for use in accordance withembodiments of the present invention.

FIGS. 4A-C illustrate asset handling in accordance with aspects of thepresent invention.

FIGS. 5A-C illustrate tile generation in accordance with aspects of thepresent invention.

FIG. 6 illustrates tile generation in accordance with aspects of thepresent invention.

FIGS. 7A-C illustrate assets and image tile generation and indexing inaccordance with aspects of the present invention.

FIGS. 8A-B illustrate a quadtree hierarchical spatial data structure andtile generation in accordance with aspects of the present invention.

FIGS. 9A-B illustrate an index table and a quadtree table for use withaspects of the present invention.

FIG. 10 illustrates a GUI for use in accordance with aspects of thepresent invention.

FIG. 11 presents a flow diagram illustrating processing of a time-basedimage request in accordance with aspects of the present invention.

FIGS. 12A-C illustrate a distributed imagery architecture in accordancewith aspects of the present invention.

FIG. 13 presents an imagery processing procedure in accordance withaspects of the present invention.

DETAILED DESCRIPTION

The aspects, features and advantages of the present invention will beappreciated when considered with reference to the following descriptionof preferred embodiments and accompanying figures. The same referencenumbers in different drawings may identify the same or similar elements.Furthermore, the following description does not limit the presentinvention; rather, the scope of the invention is defined by the appendedclaims and equivalents.

In accordance with aspects of the present invention, assets havingdifferent resolution and/or times of capture may be processed, forinstance by “blending” parts of multiple assets together in order toprovide images of a particular location or region at particular pointsin time. FIG. 1 illustrates an exemplary process 100 which produces oneor more blended images for a given date, for instance in response to auser request.

The process 100 includes block 102, where the assets (e.g., aerialimagery) are obtained. By way of example, aerial reconnaissance datesback to World War I, if not earlier. Governments have used satellitessince the 1960s to take images of the Earth. More recently, commercialsatellites have also generated images of the Earth. Assets fromdifferent sources may be collected and stored in an image database. Assuch assets may date from different times, the acquisition date for eachasset is identified as shown in block 104.

Various assets can be received from one or more sources and can have avariety of orientations. Such assets can be re-projected into a suitablecoordinate system for the map system (e.g., a geospatial coordinatesystem) and stored in one or more data structures (e.g., databasetable). The re-projected assets may then be divided into tiles which areprocessed independently, for example in a parallel processinginfrastructure. The tiles may be stored so tiles that include imageryfor geographic locations that are close to each other have a highprobability of being stored on the same machine or in the same machinecluster to reduce the overhead associated with accessing informationlocated on multiple machines. In this case, the tiles can be sized tofall within the storage constraints of the machines or a cluster ofmachines. The assets can be divided into any desired shape. A tileshape, however, typically requires less computational and/orrepresentational overhead during processing. A discussion of such tilemanipulation, including coverage masks, feathering and “minification”(e.g., resolution upsampling or downsampling) is provided in U.S. patentapplication Ser. No. 11/437,553 (“the '553 application”), entitled“Large-Scale Image Processing Using Mass Parallelization Techniques,”filed May 19, 2006, the entire disclosure of which is hereby expresslyincorporated by reference herein.

As shown in block 106, one or more “tiles” may be generated from theassets. Tiles represent a section of imagery at a particular resolutionand location. For instance, a given asset may image a certain region ofthe Earth. FIGS. 2A-2C illustrate an example of three differentoverlapping assets taken at different times for a particular region 200.In this example, FIG. 2A shows a first asset 202 having a pair ofbounding coordinates X₂, Y₁₀ and X₅, Y₅; FIG. 2B shows a second asset204 having a pair of bounding coordinates X₄, Y₆ and X₁₀, Y₃; and FIG.2C shows a third asset 206 having a pair of bounding coordinates X₀, Y₈and X₈, Y₀. The coordinates may represent latitude and longitude,Cartesian coordinates or some other geographic coordinate system. Inthis example, asset 202 is from a time T₁, asset 204 is from a time T₂,and asset 206 is from a time T₃. A user may request a tile which fallsin a region covered by multiple assets. By way of example, a first tile208 may fall within both assets 202 and 206, while a second tile 210 mayoverlap assets 204 and 206, as shown in FIG. 2D.

Tiles covering the same region need not be the same size or the sameresolution. FIGS. 2E-F illustrate a region 200′ which may be covered bydifferent sets of tiles. As shown in FIG. 2E, the region 200′ may becovered by a single tile 220 having a first resolution. As shown in FIG.2F, the region 200′ may be covered by a quartet of tiles 222. Each tile222 may have a second resolution such as a higher resolution than thefirst resolution. And as shown in FIG. 2G, the region 200′ may becovered by a set of tiles 224, for instance 16 tiles each having a thirdresolution. The third resolution may be higher than the secondresolution. Thus, in this example, tile size may decrease as resolutionor level of detail (“LOD”) increases. Alternatively, each tile 220, 222and 224 may have the same resolution, e.g., 256×256. Thus, the fourtiles 222 may have a combined 512×512 pixels, and the sixteen tiles 224may have a combined 1024×1024 pixels. Interpolation may be used togenerate the tiles 222 from tile 220. Similarly, interpolation may beused to generate the tiles 224 from tile 220 and/or tiles 222. This isknown as magnification. Conversely, finer resolution imagery may beresampled to coarser resolution by known imaging techniques. While onlythree resolutions are identified in this example, it should beunderstood that any number of resolution levels may be provided.

Returning to FIG. 1, at block 108 the LOD for each asset may be used togenerate one or more images for each tile. For instance, asset 202 mayhave a resolution of 20 cm, asset 204 may have a resolution of 50 cm,and asset 206 may have a resolution of 2 m. In a typical system,different parameters including resolution as well as image quality andimage coverage may be used to determine how a given tile is generated.

However, as more and more assets are collected, older assets may becovered up by newer imagery. For instance, asset 206 may completelyobscure asset 202 and may partially obscure asset 204. It may bedesirable to make the older imagery available so that users can see howthe surface of the Earth has changed over time. Thus, in accordance withone aspect of the invention, overlapping assets may be blended togetherbased on acquisition date, as shown in block 110. The blending resultsin tiles such as tiles 208 and 210 of FIG. 2D.

And as shown in block 112 of FIG. 1, a series of images may be producedfor each tile, for instance generating one tile image for each uniquedate when image assets intersecting the tile were originally acquired.Such tile images may be stored in an image database and/or associatedwith a data structure, as will be discussed in more detail below. Oncethe tile images have been created, they may be provided to a clientdevice, for instance in response to a user request, such as shown atblock 114.

Different architectures may be employed to achieve such results. Forinstance, FIG. 3A presents a schematic diagram of a computer systemdepicting various computing devices that can be used alone or in anetworked configuration in accordance with aspects of the presentinvention. For example, this figure illustrates a computer network 300having a plurality of computers 302, 304, 306 and 308 as well as othertypes of devices such as portable electronic devices such as a mobilephone 310 and a PDA 312. The computer processing systems may beinterconnected via a local or direct connection 314 and/or may becoupled via a communications network 316 such as a LAN, WAN, theInternet, etc. and which may be wired or wireless.

Each computer processing system can include, for example, one or morecomputing devices having user inputs such as a keyboard 318 and mouse320 and/or various other types of input devices such as pen-inputs,joysticks, buttons, touch screens, etc., as well as a display 322, whichcould include, for instance, a CRT, LCD, plasma screen monitor, TV,projector, etc. Each computer 302, 304, 206 and 308 may be a personalcomputer, server, etc. By way of example only, computers 302 and 306 maybe personal computers while computer 304 may be a server and computer308 may be a laptop. As shown in FIG. 3B each computer such as computers302 and 304 contain a processor 324, memory 326 and other componentstypically present in a computer.

Memory 326 stores information accessible by processor 324, includinginstructions 328 that may be executed by the processor 324 and data 330that may be retrieved, manipulated or stored by the processor. Thememory may be of any type capable of storing information accessible bythe processor, such as a hard-drive, ROM, RAM, CD-ROM, flash memories,write-capable or read-only memories. The processor 324 may comprise anynumber of well known processors, such as processors from IntelCorporation. Alternatively, the processor may be a dedicated controllerfor executing operations, such as an ASIC.

The instructions 328 may comprise any set of instructions to be executeddirectly (such as machine code) or indirectly (such as scripts) by theprocessor. In that regard, the terms “instructions,” “steps” and“programs” may be used interchangeably herein. The instructions may bestored in any computer language or format, such as in object code ormodules of source code. The functions, methods and routines ofinstructions in accordance with the present invention are explained inmore detail below.

Data 330 may be retrieved, stored or modified by processor 324 inaccordance with the instructions 328. The data may be stored as acollection of data. For instance, although the invention is not limitedby any particular data structure, the data may be stored in computerregisters, in a relational database as a table having a plurality ofdifferent fields and records, XML documents, or flat files. As will beexplained in more detail below, certain image-related data may be storedin flat files such as keyhole flat files (“KFF”).

The data may also be formatted in any computer readable format such as,but not limited to, binary values, ASCII or EBCDIC (ExtendedBinary-Coded Decimal Interchange Code). Similarly, the data may includeimages stored in a variety of formats such as vector-based images orbitmap images using lossless (e.g., BMP) or lossy (e.g., JPEG) encoding.Moreover, the data may include any information sufficient to identifythe relevant information, such as descriptive text, proprietary codes,pointers, references to data stored in other memories (including othernetwork locations) or information which is used by a function tocalculate the relevant data.

Although the processor 324 and memory 326 are functionally illustratedin FIG. 3B as being within the same block, it will be understood thatthe processor and memory may actually comprise multiple processors andmemories that may or may not be stored within the same physical housingor location. For example, some or all of the instructions and data maybe stored on a removable CD-ROM and others within a read-only computerchip. Some or all of the instructions and data may be stored in alocation physically remote from, yet still accessible by, the processor.Similarly, the processor may actually comprise a collection ofprocessors which may or may not operate in parallel. Data may bedistributed and stored across multiple memories 326 such as hard drivesor the like.

In one aspect, server 304 communicates with one or more client computers302, 306 and/or 308, as well as devices such as mobile phone 310 and PDA312. Each client computer or other client device may be configuredsimilarly to the server 304, with a processor, memory and instructions,as well as one or more user input devices 318, 320 and a user outputdevice, such as display 322. Each client computer may be a generalpurpose computer, intended for use by a person, having all thecomponents normally found in a personal computer such as a centralprocessing unit (“CPU”), display, CD-ROM or DVD drive, hard-drive,mouse, keyboard, touch-sensitive screen, speakers, microphone, modemand/or router (telephone, cable or otherwise) and all of the componentsused for connecting these elements to one another.

The server 304 and client computers and other devices are capable ofdirect and indirect communication with other computers, such as overnetwork 316. Although only a few computing devices are depicted in FIGS.3A and 3B, it should be appreciated that a typical system can include alarge number of connected servers and clients, with each differentcomputer being at a different node of the network. The network 316, andintervening nodes, may comprise various configurations and protocolsincluding the Internet, intranets, virtual private networks, wide areanetworks, local networks, private networks using communication protocolsproprietary to one or more companies, Ethernet, WiFi, Bluetooth andHTTP.

Communication across the network, including any intervening nodes, maybe facilitated by any device capable of transmitting data to and fromother computers, such as modems (e.g., dial-up or cable), networkinterfaces and wireless interfaces. Server 304 may be a web server.Although certain advantages are obtained when information is transmittedor received as noted above, other aspects of the invention are notlimited to any particular manner of transmission of information. Forexample, in some aspects, the information may be sent via a medium suchas a disk, tape, CD-ROM, or directly between two computer systems via adial-up modem. In other aspects, the information may be transmitted in anon-electronic format and manually entered into the system.

Moreover, computers and client devices in accordance with the systemsand methods described herein may comprise any device capable ofprocessing instructions and transmitting data to and from humans andother computers, including network computers lacking local storagecapability, PDA's with modems such as PDA 312 and Internet-capablewireless phones such as mobile phone 310.

As shown in FIG. 3A, the network 300 may also include an image database332 and/or a map information provider 334. The image database 332 andthe map information provider 334 may be directly or indirectly coupledto server 304. In an alternative, the image database 332 and the mapinformation provider 334 may be part of or otherwise logicallyassociated with the server 304. The image database 332 may store data330 in one or more KFFs. The map information provider 334 may obtainassets and other information, including satellite data, aerialphotographs, digital maps, elevation data, GPS coordinates, etc. fromone or more sources (not shown). Examples of an image database and a mapinformation provider are provided in co-pending and jointly owned U.S.patent application Ser. No. 11/762,049 (“the '049 application”),entitled “Markup Language for Interactive Geographic InformationSystem,” filed Jun. 12, 2007 and published as U.S. Patent PublicationNo. 2008/0016472 on Jan. 17, 2008, the entire disclosure of which ishereby expressly incorporated by reference herein. Furthermore, eachclient device (e.g., computers 302, 306 and 308, as well as mobile phone310 and PDA 312), may include or run application software such as ageospatial browser, which may include a mapping module, as disclosed inthe '049 application.

As discussed above with regard to FIGS. 2A-2D, different overlappingassets may be imaged at different times for a particular region ofinterest. Thus, when preparing a given tile encompassed by multipleassets, different options are available. For instance, one may “rank”the assets based on resolution. Here, the highest resolution assets maybe placed on a top or frontmost layer, while the next higher resolutionasset may be placed in a next lower layer, and so on in order ofdecreasing resolution. The layers may be blended in this manner toprovide for a “best” resolution based upon the available assets.

The example set forth above in FIG. 2D presents such a scenario. Here,tiles 208 and 210 are formed using the best available asset resolution.Thus, as shown in the figure, the tile 208 may be comprised solely of aportion of the asset 202, while the tile 210 may comprise a portion ofthe asset 204 (shown as the lower half of the tile) and a portion of theasset 206 (shown as the upper half of the tile.

However, tiles may be formed based on criteria other than resolution. Aparticularly suitable criterion is by date of acquisition of theasset(s). For instance, in many situations a user may choose to view aregion of interest as it looked at one or more points in time.Evaluating how the landscape evolves is often relevant to urban planning(e.g., how has a city expanded over time), ecology (e.g., has the sizeof wetlands shrunk due to construction) and home buyers (e.g., how manyhomes in the neighborhood have been renovated in recent years), to namea few. Thus, according to one aspect of the present invention, assetsmay be blended to produce multiple outputs (e.g., tiles) for a givenlocation, where each output represents the appearance of that locationat a particular time.

In the example in FIGS. 2A-C, the asset 202 is from time T₁, the asset204 is from time T₂, and the asset 206 is from time T₃. Suppose thattime T₁ is the earliest and time T₃ is the most recent. In this case,one could produce at least three blended views. Examples of such viewsare presented in FIGS. 4A-C. As shown in FIG. 4A, the view 400 at timeT₁ includes only asset 202. The view 402 at time T₂, shown in FIG. 4B,includes both asset 202 and asset 204. Here, a combination of the twoassets has asset 204 overlying asset 202, even though asset 202 mayinclude higher resolution imagery. And as shown in FIG. 4C, the view 404incorporates all three assets 202, 204 and 206, with the most recentasset 206 overlying both asset 204 and asset 202.

If a user requests a view covered by a tile comparable to tile 208 ofFIG. 2D, the resultant tile provided will depend on the time ofinterest. For instance, as shown in FIG. 5A, at time T₁ tile 500 will beprovided, where this tile is equivalent to tile 208 of FIG. 2D as bothare derived from asset 202. As shown in FIG. 5B, at time T₂ the tile 500is also provided, as asset 204 does not encompass this region. Incontrast, as shown in FIG. 5C, at time T₃ tile 500′ is provided, wherethis tile comprises imagery from asset 206 but preferably not from asset202.

Similarly, if a user requests a view covered by a tile comparable totile 210 of FIG. 2D, the resultant tile provided will depend on the timeof interest. Here, tile 210 is illustrated in broken lines forreference. In this case, at time T₁ (see FIG. 5A) no tile will beprovided because asset 202 does not encompass the region of interest. Asshown in FIG. 5B, at time T₂ tile 502 is provided. In this example, onlya portion of tile 210, namely tile 502, is available due to the coverageof asset 204. In this case, the display may not provide an image for theunavailable portion of tile 210, may indicate to the user that no datais available for that portion of the region of interest, or may providea default image that may or may not have a date associated with it. Incontrast, as shown in FIG. 5C, at time T₃ tile 502′ is provided. Thistile comprises imagery from asset 206 but preferably not from asset 204.

FIG. 6 illustrates another case involving assets 202, 204 and 206. Aswith the previous examples, asset 202 is the oldest asset, asset 204 isthe next oldest asset, and asset 206 is the most recent asset. For easeof illustration, the three assets are presented with asset 204 overliesasset 202 and with asset 204 being partly transparent to show theearlier asset. Similarly, asset 206 overlies assets 202 and 204, and ispartly transparent to show both earlier assets.

In the present case, a new region of interest illustrated 600 is shown.Here, region of interest 600 is encompassed by all three assets 202, 204and 206. Thus, as shown by the dashed arrow, at time T₁ a first tilewould be formed by the overlapping portion of asset 202. As shown by thedashed arrow, at time T₂ a second tile would be formed by theoverlapping portion of asset 204. And as shown by the dashed arrow, attime T₃ a third tile would be formed by the overlapping portion of asset206.

In an alternative, it is possible to use imagery from an underlyingasset to fill in any gaps or address any defects in the overlying assetof the desired point in time. However, this is preferably done onlyalong the edges of the overlying asset. This is because effects ofspatial and/or color misregistration between assets can be magnified,resulting in a poor quality image or tile. Feathering the edges ofadjacent asset imagery makes the edges less prominent and distracting.

In accordance with an aspect of the present invention, blending ofadjacent and/or overlapping tiles is done using asset acquisition dateas the primary criterion blending criterion. FIG. 7A presents an examplewhere six assets (A-F) are obtained at six different times (T₁, . . . ,T₆). Each asset is associated with a level of detail, LOD, rangingbetween 1 (e.g., lowest resolution) and 6 (e.g., highest resolution). Inthe present case, none of the assets A-F completely covers the tile ofinterest.

FIG. 7B illustrates how multiple tile images are generated for the tileof interest when multiple assets of different acquisition dates areavailable. Thus, as shown in the rightmost column in this figure, sixdifferent tile images may be generated. As presented in the figure, thesymbol “+” means that the asset to the right of the + overlays the assetto the left of the + when blended. For instance, while at time T₁ onlyasset B is used to generate a given tile image, at time T₂ assets B andC are blended to generate another tile image. In this case, as asset Bwas acquired at time T₁ and asset C was acquired at time T₂, asset Coverlays asset B in the blended tile image. Similarly, for time T₃, thethree assets B, C and E are used, with asset E overlying C and Coverlying B. Additional tile images for times T₄, T₅ and T₆ aregenerated similarly.

A user who is interested in viewing this particular tile of interest hasthe option to view the tile at six different points in time. By way ofexample, the user interface of a client device (e.g., computer 302, 306or 308, mobile phone 310 or PDA 312 of FIG. 3A) may provide the userwith a slider bar, radio buttons or other actuator to select aparticular point in time. Thus, if the user selects time T₁, then thetile image based solely on asset B is shown. If the user selects timeT₄, then the tile image based on the combination of assets B, C, E and D(blended in the manner shown in the first row) is provided. And if theuser selects time T₆, then the tile image based on the combination ofassets B, C, E, D, A and F (blended in the manner shown in the firstrow) is provided.

If asset acquisition time was the only criterion, then the first row ofFIG. 7B would provide all of the necessary tile images for the tile ofinterest. However, in accordance with another aspect of the presentinvention, further tile images may be generated based on LOD and/orother criteria. In the present figure, additional tile images aregenerated in view of LOD. As shown in the second row (LOD=2), fewer tileimages may be generated due to the omission of asset A, which has a LODof 1. The tiles generated at LOD 2 may be of different size and/orresolution than the tiles generated at LOD 1. For instance, while thetile image(s) for time T₄ having an LOD of 2 may be generated using thesame assets (B+C+E+D) as the tile image(s) for time T₄ having an LOD of1, more tiles (e.g., sub-tiles) of higher resolution may be employed.

In the example of FIG. 7B, no new tile image need be generated in thiscase for time T₅, as asset A is the only asset for that point in time.Therefore, should a user request a tile of interest at time T₅ having aminimum resolution of at least 2, the tile image generated for time T₄or another point in time may be displayed, or the client device maymagnify the data. Furthermore, for time T₆, in one example only assetsB, C, E, D and F are employed at this resolution level. In anotherexample, a blend of higher resolution images may involve a coarserresolution asset such as asset A. Thus, a tile of interest for time T₆may include data magnified from the original resolution of asset A. Inone variation, only coarser levels from older assets will be used.Different combinations of assets may be generated for differentresolution levels as illustrated in FIG. 7B.

In another variation, the oldest or earliest available asset may bedesignated to fill in empty or otherwise deficient spots in tiles. Thisearliest or “baseline” asset may be used as a background layerencompassing the entire tile. This base asset (e.g., asset 0 from timeT₀) may be magnified (wherein a magnified asset is represented by M_(X))as needed in a given blend. In the following example, it is assumed thatmagnifying by one or two levels is sufficient to make an assetcompletely cover a tile. Thus, a modified version of the table in FIG.7B may be as follows:

LOD Time-Based Tile(s) 1 T₁ = M₀ + B; T₂ = M₀ + B + C; T₃ = M₀ + B + C +E; T₄ = M₀ + B + C + E + D; T₅ = M₀ + B + C + E + D + A; T₆ = M_(A) +B + C + E + D + F 2 T₁ = M₀ + B; T₂ = M₀ + B + C; T₃ = M₀ + B + C + E;T₄ = M₀ + B + C + E + D; T₆ = M₀ + M_(A) + B + C + E + D + F 3 T₂ =M_(B) + C; T₃ = M_(B) + M_(C) + E; T₄ = M_(B) + M_(C) + E + D; T₆ =M_(B) + M_(C) + E + D + F 4 T₂ = M_(B) + C; T₃ = M_(B) + M_(C) + E; T₄ =M_(B) + M_(C) + E + D; T₆ = M_(B) + M_(C) + E + D + F 5 T₃ = M_(B) +M_(C) + E; T₄ = M_(B) + M_(C) + E + D; T₆ = M_(B) + M_(C) + E + D + F 6T₆ = M_(C) + M_(E) + M_(D) + F

In the above table, for an LOD of 3, if magnifying asset B by one levelis not sufficient to cover the tile, then the time-based tiles would bederived by the following blends: T₂=M₀+M_(B)+C; T₃=M₀+M_(B)+M_(C)+E;T₄=M₀+M_(B)+M_(C)+E+D; T₆=M₀+M_(B)+M_(C)+E+D+F. Also, as shown, noadditional dates are triggered by magnified assets at higher than theirnative resolution. Thus, the LOD of 5 does not have T₁=M_(B).

As discussed above with regard to FIG. 1, one or more images may begenerated for each tile by blending available assets (see block 112).For a large collection of overlapping assets there are potentially manypossible blends. For instance, in a case where there are three assets(e.g., A, B and C), it is possible to blend 15 combinations (e.g., A, B,C, AB, AC, BA, BC, CA, CB, ABC, ACB, BAC, BCA, CAB and CBA). Each blendmay form a different output image. The potential large number of blendsmay make displaying and navigation difficult. Therefore, in accordancewith an aspect of the present invention, a criterion which variesmonotonically from blended image to blended image is used to order andreduce the quantity of blended images.

One such criterion is time. The tile images may be primarily or solelybased on the acquisition date of each asset. Optionally, other criteriasuch as LOD may be employed to generate tile images. Generating tileimages focusing on acquisition date enables the architecture to providea user with the option to view a region of interest at various points intime. Thus, the user may see images showing how the region of interesthas changed over time.

In accordance with another aspect of the invention, because LOD is notprimarily determinative of the blending order of assets, there may be noneed to eliminate poor quality sections of images. As discussed above,while it is possible to use imagery from an underlying asset to fill inany gaps or address any defects in the overlying asset, effects ofspatial and/or color misregistration between assets can be magnified.Therefore, in this case, unless there is a direct collision ofacquisition dates (e.g., two assets have the exact same time ofacquisition), it is preferred not to eliminate poor quality sections ofan asset's image. Alternatively, if such direct collision does occur, ahybrid blending scheme incorporating LOD or other criteria may beemployed.

Another possible issue with asset acquisition is that different assetsmay be received, e.g., by map information provider 334, from multiplevendors. Each vendor may have a different may of reporting assetacquisition dates. Some assets may have no date at all. Others may becomposed from images that were acquired over a period of several days,months or years. Some assets may have dates that contain just the year,or just the year and month. And some assets may have an acquisition timewhich is accurate to the second. In one embodiment, all acquisitiongranularities will be accepted and sorted accordingly. In anotherembodiment, the range of acquisition may be narrowed to a time interval.In this case, the end of the time interval may be used as theacquisition date. For instance, if an asset is composed of images takenover the course of Jun. 1, 2008 through Jun. 30, 2008, then Jun. 30,2008 would be the acquisition date associated with that asset.

In accordance with another aspect of the present invention, assets maybe indexed in accordance with the acquisition date and/or image tilesmay be indexed by most recent acquisition date resulting from theblending operation. Given the available acquisition information for tileimages, it is desirable to associate such information with a common dateformat. By way of example, the date may be in the following format:YYYYMMMDDHHMMSS. In one example, each asset may be indexed withacquisition date (e.g., YYYYMMMDDHHMMSS format), location (e.g., X, Ypairs, latitude and longitude format, etc) and LOD, such as shown inFIG. 7C. Each blended image tile generated for a given tile/region ofinterest may also be stored in accordance with such parameters.

Once tiles have been generated and/or blended for different times ofinterest, the tiles should be stored and indexed in a manner enablingeasy access and/or manipulation. As discussed above, multiple imagetiles for a given region of interest may be associated with a resultantacquisition time, blended level of detail and location.

In one example, image tiles and associated data are stored in a“quadtree” structure. FIGS. 8A and 8B illustrate an exemplaryhierarchical spatial data structure 800 and its application to a tile802 of imagery. In the example shown in FIG. 8A, the hierarchicalspatial data structure 800 is a quadtree. A quadtree is a rooted treestructure where every internal node includes four child nodes. In theexample shown, a root node R includes child nodes A, B, C and D. Each ofthe internal child nodes A and C has four child nodes. For example,internal child node A has four child nodes: A₁, A₂ A₃, and A₄. Likewise,internal child node C has four child nodes: C₁, C₂, C₃ and C₄. Followingthis pattern, the internal child node A₄ has four child nodes: A_(4,1),A_(4,2), A_(4,3) and A_(4,4). While only two levels of the quadtree datastructure 800 are shown, the quadtree data structure 800 can have anydesired number of levels depending on the application. The quadtree datastructure 800 is a well-known hierarchical data structure that has avariety of useful properties. Quadtree data structures are described inFoley et al., “Computer Graphics: Principals and Practice Second Editionin C:” Addison-Wesley (1996) (see chapters 12 and 15), which isincorporated by reference herein in its entirety.

The quadtree data structure 800 is particularly well-suited for storingimagery and associated metadata. In the example shown, the root R of thequadtree data structure 800 can be mapped to tile 802, which is shown inFIG. 8B. The tile 802 can be generated as described elsewhere herein,and can also be generated as described in U.S. patent application Ser.No. 11/473,461 (“the '461 application”), entitled “Hierarchical SpatialData Structure and 3D Index Data Versioning for Generating Packet Data,”filed Jun. 22, 2006, the entire disclosure of which is hereby expresslyincorporated by reference herein. The tile 802 can be further dividedinto four quadrants A, B, C, D, each of which can be mapped to childnodes A, B, C and D of the quadtree data structure 800. Each of the fourquadrants A, B, C and D can be further divided into four quadrants andso forth. Thus, there can be a direct mapping between nodes in thequadtree data structure 800 and quadrants in the tile 802. In theexample shown, the quadrants A_(4,1), A_(4,2), A_(4,3), and A_(4,4) inthe tile 802 map to nodes A_(4,1), A_(4,2), A_(4,3), and A_(4,4),respectively, in the quadtree data structure 800. The nodes of thequadtree data structure 800 are referred to herein as “quadnodes.”

A quadnode plus one or more levels of descendents are referred to hereinas “quadsets.” Data pertaining to quadtrees, quadnodes and quadsets maybe stored in a database such as a KFF database. One discussion of a KFFdatabase structure may be found in U.S. Pat. No. 7,225,207, entitled“Server for Geospatially Organized Flat File Data,” the entiredisclosure of which is hereby expressly incorporated by referenceherein. While the description above is in reference to quadtree datastructures, other hierarchical spatial data structures can be used inany of the disclosed implementations, such as octrees, k-d-trees,b-trees, bv-trees and BSP-trees.

Tile imagery and metadata may be associated with quadnodes of a quadtreedata structure. The locations of the files that store the data for eachquadnode can be stored in an index table 900, as shown in FIG. 9A. Insome implementations, the “data location” column in the index table 900can include numbers rather than filenames to reduce storage or memoryrequirements. The numbers can be used to index into a table of files atthe datacenter. In the example shown, the index table 900 can include arow for each of N quadnodes in the quadtree data structure or quadtreetable 902 of FIG. 9B. The index table 900 may also include a separatedate indicator for time of creation (e.g., acquisition date) associatedwith the imagery data. For instance, a row key may be extended from onlya location identifier to a location plus date identifier.

The contents of each row in the index table 900 may include a dataversion number and file location (e.g., a pathname plus a filename)where the quadnode data is stored. Quadnode data can include any desireddata, including but not limited to imagery, terrain and vector data, aswell as acquisition date. Vector data can be overlaid on the imagery atdesignated locations for various levels or layers of detail. Someexamples of vector data include information related to gas stations,restaurants, points of interest and the like. The files can be part of aglobal file system, such as a KFF file structure.

Each row of the index table 900 may be read by a mapping and datareduction process and written to the quadtree table 902. In someimplementations, each row of the quadtree table 902 is a quadset andincludes data associated with the quadset (e.g., quadtree data for a setof quadnodes). The name of the row can be the root node of the quadset.For example, the first row of the quadtree table 902 could include dataassociated with quadset 1, the second row could include data associatedwith quadset 2 and so forth. The index table 900 and quadtree table 902can be stored on any suitable computer readable medium (e.g., hard disk,memory, optical disk, etc.).

Due to storage, processing and other factors, creating and maintainingnew databases can be resource and cost intensive. In some situations, animagery database may already exist with a single tile for each region ofinterest. For instance, a particular tile may have been blended basedsolely on LOD and stored in the imagery database. It is possible thatthis particular tile has the same view as an image tile which wouldgenerated by the acquisition date-based processing disclosed herein. Inthat case, it is desirable to leverage the existing database to avoidduplicative storage of such tiles. Therefore, while a new database mayinclude a set of new assets and/or tile images with predeterminedacquisition dates, the existing database and the new database may belinked, (e.g., via a pointer added to an index table) indicating whichdatabase stores a given image tile. Additional data may be linked withthe preexisting tile from the existing database to associate theparticular tile with an acquisition date. Such linked databases wouldhelp to leverage any existing imagery database.

In accordance with an aspect of the present invention, one or moredatacenters may store and maintain imagery data which is provided toclient devices upon request. By way of example, a primary datacenter andone or more distributed datacenters may be provided. FIG. 12Aillustrates one such distributed architecture 1200 including primarydatacenter 1202 and distributed datacenter 1204 connected to a network1206, which in turn may couple one or more user devices, e.g., devices308, 310 and 312 of FIG. 3A. As shown, primary datacenter 1202 includesa database 1208, which may maintain image data and/or packfiles as willbe explained in more detail below. The distributed datacenter 1204 mayinclude one or more databases such as databases 1210 and 1212. Thesedatabases may store image-related data dependent upon differentcriteria, as will be explained in more detail below.

FIG. 12B illustrates a blending and updating scenario 1220 whichincorporates assets 1222 and a tiles table 1224 formaintaining/organizing tiles. The assets and tiles table may be storedor otherwise maintained in database 1208 of the primary datacenter 1202.At blending block 1226 the assets and/or tiles table are blended asdescribed herein. The tiles table may or may not include time-basedinformation. The tiles table may be indexed by location and level ofdetail. In the case where some or all of the tiles include time-basedinformation, the tiles table may also be indexed by date.

In one example, blending per block 1226 occurs in primary datacenter1202. In this example each execution of the blending process, whether ornot time-based information is employed, produces a version or “epoch.”The resulting blends may be placed into packfiles 1228 for thatversion/epoch. Upon creation of the packfiles 1228, fingerprintsassociated with the tiles may be updated per block 1230. As used herein,a fingerprint is a hash of the information that is used to create atile. The hash/fingerprint may be used to answer whether the tile beingproduced with the current assets and other parameters is the same as onethat was previously produced.

Consider an example for a single location and LOD during the blendingprocess. In this example, time-based assets may be employed inconjunction with existing non-time-based imagery. In accordance with anaspect of the invention, assets intersecting a given tile are sorted bydate and are blending in order from oldest to newest. As each dated tileis processed, several fingerprints may be considered (if they exist).For instance, a non-time-based fingerprint from the tile table mayrepresent a released non-time-based tile. A time-based fingerprint fromthe tile table may represent a released and dated tile. And a newlygenerated time-based fingerprint may be associated with the dated tilebeing processed. When a new fingerprint is generated for a dated tileundergoing processing and is compared to an existing tile tablefingerprint, numerous possibilities may occur, as shown in the followingtable.

Non-Time-Based Fingerprint Time-Based Fingerprint Action Not equal Notequal Blend Not equal Equal Skip (1) Equal Not equal Share Equal EqualShare Equal Previously deleted or shared Skip (2) Not equal Previouslydeleted or shared Blend (3) Previously released & deleted Previouslydeleted or shared Blend (4) Previously released & deleted Equal Skip (1)Previously released & deleted Not equal Blend Doesn't exist Blend

Note “(1)” in the table indicates that the tile already exists in aprevious time-based version, cannot be shared with a non-time-based tiledatabase, and does not need to be reblended. Note “(2)” in the tableindicates that the tile has been previously shared, but no changes havebeen made to a non-time-based asset, so the tile can be skipped orre-shared. Note “(3)” in the table indicates that the tile waspreviously shared with the non-time-based tile database, but thenon-time-based tile has been updated and can no longer be shared andthus is to be blended. And note “(4)” in the table indicates that a tilewas previously deleted from the tile databases. In this case, when an“in-process” dated tile (e.g., a time-based tile currently beinggenerated) has assets associated with it, then it should be blended. Asshown in FIG. 12B, after a fingerprint is updated per block 1230,information regarding the updated fingerprint may be provided to thetile table 1224.

The packfiles 1228 formed as a result of blending per block 1226 mayhave one or more entries therein. A given entry in a packfile 1228 maycontain a tile, an indication that a tile is shared with a database suchas the non-time-based tile database, or a deletion marker indicatingthat the entry should be deleted. Each entry in a packfile 1228 may bedated when the packfile 1228 contains time-based data. Each entry in aspecific packfile 1228 that is a tile may also contain a new/currentfingerprint for that tile.

The packfiles 1228 may be sent to other datacenters, such as distributeddatacenter 1204, for further processing. This may be done by issuing thepackfiles on disk or other recording medium, or otherwise outputting thepackfiles to distributed datacenters 1204. Alternatively, the primarydatacenter 1202 may write tiles and/or other image-related data to theserving datacenter(s) 1204 without issuing packfiles. FIG. 12Cillustrates an indexing and quadtree generation scenario 1240 which mayoccur when packfiles 1228 are sent to distributed datacenter(s) 1204. Inthe present example, indexing and quadtree generation processes may berun/performed at the datacenter(s) which will provide the data to clientdevices. Alternatively, such operations could be performed in primarydatacenter 1202. In this case, the index and quadtree tables aredistributed from the primary datacenter 1202 to the distributeddatacenter(s) 1204. As shown in the figure, packfiles 1228 may bedistributed or segregated depending upon whether they contain time-baseddata or not. For instance, time-based packfiles 1228 a may be managedseparately from non-time-based packfiles 1228 b.

After packfiles 1228 have been copied to a given distributed datacenter1204, the indexing process scans the packfiles and updates the indextable (e.g., table 900) with one entry per tile. For instance, in block1242, the indexing process for time-based packfiles 1228 a is performed,while the indexing process for non-time-based packfiles 1228 b isperformed in block 1244. The index table key for time-based index table1246 includes the date for the tile, while the index table key fornon-time-based index table 1248 does not include date information.

After indexing, as shown by blocks 1250 and 1252, a quadtree builderprocess (quadtree generation) collects information from nearby locationsand LODs in a hierarchical fashion to produce updates to the respectivequadtree tables. As shown by dotted line 1254, for the time-basedprocessing, reference may also made to the non-time-based index table1248 to obtain the current version number for shared tiles, which isincluded in the quadtree node when there is a shared tile. For anylocation and level, only one dated tile is shared from thenon-time-based imagery database. The result of quadtree generationblocks 1250 and 1252 are quadtree packet (“QTP”) tables 1256 and 1258,respectively.

The packfiles 1228 a and 1228 b may also be processed to update datatables at the distributed datacenter 1204, as shown by blocks 1260 and1262, respectively. Data tables 1264 and 1266 contain actual image datafor each tile, keyed by location and LOD. For time-based data, thetime-based data table 1264 also include the dates associated with therespective image data. Serving tables (e.g., the data and QTP tables)may be provided to one or more servers. For example, there may be atime-based imagery server 1268 and a non-time-based imagery server 1270which communication with client devices via network 1272. While twoimagery servers are shown, it should be understood that a single imageryserver may handle both time-based and non-time-based imagery.Alternatively, multiple servers may be employed to manage both types ofimagery.

As the serving tables (e.g., data and QTP tables) can contain multipleversions, existing clients may not be aware of the update process forthese tables, as the updates may be for a new version. Clientsrequesting time-based data with references to shared non-time-basedtiles may still obtain such tiles from a server which manages thenon-time-based tiles during and after a non-time-based data push.

After a version is made “live” or active and available, the fingerprintsin the packfiles for that version may be copied back into the tilestable. The updated fingerprints for that version represent the active,released state of the tiles.

An example of a method incorporating such processing is presented inflow diagram 1300 of FIG. 13. In block 1302, a new non-time-basedimagery version is blended. In block 1304, non-time-based versionpackfiles are distributed, e.g., to one or more distributed datacenters1204. The old version may still be made available for clients requestingtime-based imagery. The non-time-based version packfiles are indexed atblock 1306 and the non-time-based data and quadtree packet tables areupdated in block 1308. The new non-time-based version is made live inblock 1310. The non-time-based fingerprints are updated in the tilestable per block 1312.

In block 1314, a new time-based imagery version is blended. Currentnon-time-based fingerprints may be used to detect sharing of tiles. Inblock 1316, time-based version packfiles are distributed, e.g., to oneor more distributed datacenters 1204. The time-based version packfilesare indexed at block 1318 and the time-based data and quadtree packettables are updated in block 1320. The new time-based version is madelive in block 1322. The time-based fingerprints are updated in the tilestable per block 1324.

At block 1326, a “garbage collection” process may remove older versionsof non-time-based tiles which are no longer referenced. At block 1328,older versions of time-based tiles that have been updated may beremoved. This may be done, for instance, when color parameters havechanged. If no garbage collection is performed, it is possible to pushtime-based or non-time-based imagery to a time-based imagery database ora non-time-based imagery database, respectively, more than once withoutpushing the other database.

It should be understood that while flow diagram 1300 presents blocks ina certain order, the procedures/operations which are not dependent onthe results of other procedures/operations may be performed in adifferent order and/or in parallel with other blocks. By way of example,blocks 1314-1322 may be performed prior to or concurrently with blocks1302-1312.

Another aspect of the invention pertains to communication between clientdevices and the server or other device which provides imageryinformation. As noted above, a given client device may include or runapplication software such as a GUI implementing a geospatial browser,which may include a mapping module.

FIG. 10 illustrates one particular embodiment of a geospatial browserGUI 1000 for use in accordance with aspects of the present invention.The GUI geospatial browser 1000 includes a display window 1002 fordisplaying a 2D or 3D map, as well as a text input field 1004 forentering location information such as latitude and longitude, an addressand/or zip code, or the name of a well-known site (e.g., “LincolnMemorial” or “Area 51”). The GUI 1000 may include a number of modes inwhich it can operate, including Fly To mode, Local Search mode, andDirections mode, as shown by mode select buttons 1006, which is part ofthe geospatial browser main menu 1008. A discussion of the Fly To mode,Local Search mode and Directions mode may be found in the '049application.

GUI 1000 may also include a zoom control 1010 for adjusting the viewingaltitude, a tilt control 1012 for adjusting the viewing angle, rotationcontrols 1014 for rotating the view left and right, and/or a set ofpanning controls 1016 to view areas of the 2D or 3D map to the left,right, top or bottom of the display window.

GUI 1000 may also includes a places control 1018, which allows the userto organize saved data in a Places panel in a way similar to how a userwould organize files and folders on a computer's hard drive. Forexample, the places control 1018 allows the user to create folders,reorder placemarks or folders, rename a placemark or folder,remove/delete a placemark or folder, and empty a folder's contents.Also, the user can select (e.g., check box or other such GUI controlmechanism) various places designated in the places control 1018, andthen select a “play” function button (lower right of places control 1020panel) so that a virtual tour of those selected places may then bedisplayed in the window 1002. Stop and pause functions can also beprovided to give the user more control over a virtual tour.

GUI 1000 may also includes the layer control 1020, which provides avariety of data points of geographic interest (e.g., points of interest,as well as map, road, terrain, and building data) that a user can selectto display over the viewing area. In the example shown in FIG. 10,exemplary commonly used layers are available on the Navigation panel(e.g., Lodging, Dining, Roads, Boarders, Terrain, and 3D Buildings) anda full list of layers is available in the Layers panel (e.g., NationalGeographic Magazine articles relevant to a particular area, Golfcourses/ranges, Coffee Shops, Community Sites, earthquake epicenters,etc).

GUI 1000 of this example may also display image data 1022 in the lowerportion of the display window 1002, including pointer/cursor coordinates(e.g., lat/lon/altitude), streaming percentage completion, and eyealtitude (e.g., feet). The GUI 1000 may further includes print and emailcontrols 1024 (so as to allow for printing and emailing of locationsand/or images). Also, the GUI 1000 optionally includes an addplacemark/folder/network link control 1026, which allows the user tocreate or otherwise add new placemarks, folders, and/or network links.

The geospatial browser main menus 1008 may include the File menu (e.g.,functions such as Open, Save, Save As, Email/Email View, Share withOnline Community, Print, Logout), Edit (e.g., includes functions such asFind in Places, Find Next, Find Prev, Copy, Snapshot View, Past Delete,Rename, Refresh, Apply Style Template, Delete Content, Save to MyPlaces, Clear Search History, and Properties), View (e.g., includesfunctions and selectable display features such as Full Screen, ViewSize, Compass, Status Bar, Lat/Lon Grid, Overview Map, and Play Tour),Add (e.g., includes functions to allow the addition of Placemarks,Folders, Image Overlays, and Network Links), Tools (e.g., includesselectable tools such as Navigation panel, Places panel, Layers panel,Measuring tool, and Web Search panel), and Help (e.g., includes accessto online help center and other informative sources). Note that the addplacemark/folder/network link control 1026 can be configured to providemenu options that correspond to the options in the Add menu of thegeospatial browser main menus 1008. Further note that various places andlayers of the Places and Layers panels can be expanded (or condensed) toshow additional (or fewer) sub-places and sub-layers (e.g., click GUIarrow pointing at place/layer label to expand or showsub-places/sub-layers, or click GUI arrow pointing down to condense orhide sub-places/sub-layers).

In accordance with aspects of the present invention, the GUI 1000 alsoincludes date-related options. For instance, one or moreactuators/selectors 1028 may enable the user to select or deselecttime-based imagery display. If selected, the user may employ an actuatorsuch as slider 1030 to set a date of interest for the map. Anotheractuator 1032 may enable the user to choose to view multiple maps in aside by side or tiled arrangement, wherein the different maps show thesame region of interest at different points in time. In this case, theuser may employ actuator(s) 1034 to select particular dates to map, ormay use text input field 1004 to type dates or a date range. In analternative, zoom control 1010 or another control may enable the user to“zoom” or “pan” between maps for different timeframes depending uponwhether time-based imagery display has been enabled via actuator 1028.

Numerous GUI configurations and underlying functionalities will beapparent in light of this disclosure, and the present invention is notintended to be limited to any one particular configuration. Thedisplayed 2D or 3D maps can be manipulated using the GUI 1000. The GUI1000 can be used to reposition the current map view, for example, byclicking and dragging in the display window 1002. A user may also selecta geographical location or time by double-clicking on it within thedisplay window 1002.

When a user selects a time-based map with GUI 1000, the user may beprovided with information pertaining to the dates or range of dates forwhich imagery is available. For instance, the user may select a regionof interest (e.g., San Francisco), and a query may be generated indisplay window 1002 informing the user that maps are available for thefollowing dates: 1980, 1990, 2000, 2007 and 2008. The user is then giventhe option of selecting one or more maps based on dates as discussedherein. Alternatively, the user may choose to view a map withoutreference to a particular date, e.g., showing merely the highestresolution available.

In the case where the user desires to view a time-based map, data may bepassed to the client device concerning which image tiles are availablefor specific dates. The client device will then be able to request animage tile(s) in accordance with a date(s) selected by the user. Tilesmay be provided based on the resolution as chosen by the user (if any).

Upon request, an imagery server at a distributed datacenter provideshierarchical “table of contents” data, quadtree packets, to a clientdevice. When viewing a region of the Earth, the client device requeststhe quadtree packets for that region. For each tile in view, thequadtree packet indicates which tiles have imagery available, as well asother layers like terrain and vector data. This enables the clientdevice to know exactly which requests for imagery to make. Each packetmay also contain the table of contents data for several hundred tiles.This reduces the number of round trips required to get the table ofcontents data. The entry for each tile in a quadtree packet is referredto as a quadnode because it is a node in the quadtree.

In the time-base imagery database, the data in each quadnode may includea list of times for which blended images are available for that tile.This allows the client device to minimize its requests for imagery.Thus, when the user of the client device selects a new time in theinterface, the client device need only request images where the new timecorresponds to an image different from the one it is already displaying.

For example, a client device operable to handle time-based imagery mayrequest quadtree packets for the current location and level from aserver, such as the server 1268 of FIG. 12C based on the current view ofthe Earth. Using the date information in the quadtree nodes, the clientdevice may then request the appropriate dated tiles for the location andlevel from the server. If any needed tile is shared the client devicemay redirect the request to a non-time-based server, such as the server1270 of FIG. 12C, instead. Tiles may be cached in the client device on aper-server basis, so requests for shared tiles can be satisfied by thenon-time-based server cache. The client cache may also be indexed bydate for time-based tiles, so requests for dated tiles in the samelocation and level can be satisfied by the time-based server cache.

An exemplary map generation process 1100 is described below with regardto FIG. 11. First, as shown in block 1102, the client device (or server)receives a request from a user for an image associated with ageographical location at a point in time prior to the time of therequest. As shown in block 1104, one or more image tiles are obtainedwhich correspond to the time-based request. The image tiles which areobtained are selected from a number of image tiles which include imagetiles associated with images captured before and after the requestedpoint in time. Then, as shown in block 1106, in response to the request,the client device (or server) provides an image (e.g., derived fromapplicable image tiles) associated with the requested geographicallocation. In this case, the different images tiles associated with theimage provided to the user are limited to images (e.g., assets) capturedprior to the requested point in time.

In order to speed up processing, minimize network traffic and serverload, the client device may store a cache of image tiles locally. Thecache may include tile date as part of a key to allow multiple differentimages for the same tile location and resolution.

While the client may store an image cache accessible by tile date, inaccordance with another aspect of the present invention blending isdesirably performed at a server, e.g., server 304. Here, the image tilesfor a region of interest may be stored in an imagery database such asdatabase 332 of FIG. 3A. While it is possible to send some or all of theimagery database to a client device and have the client device performacquisition time based blending, such processing is computationallyintensive and better performance may result from having a server ormultiple processing devices (e.g., operating in a parallel processingmanner) perform such processing, including blending and/or compression,and transfer resultant image tiles based on a requested region ofinterest.

Although the invention herein has been described with reference toparticular embodiments, it is to be understood that these embodimentsare merely illustrative of the principles and applications of thepresent invention. It is therefore to be understood that numerousmodifications may be made to the illustrative embodiments and that otherarrangements may be devised without departing from the spirit and scopeof the present invention as defined by the appended claims. Furthermore,while particular processes are shown in a specific order in the appendeddrawings, such processes are not limited to any particular order unlesssuch order is expressly set forth herein.

The invention claimed is:
 1. A computer-implemented method comprising:receiving, at one or more computing devices from a client computingdevice, a request for three-dimensional data, the request identifying ageographic location; providing, by the one or more computing devices,information identifying a set of available points in time based on theidentified geographic location; receiving, at the one or more computingdevices from the client computing device, information identifying apoint in time in the set of available points in time; in response toreceiving the information identifying the point in time, selecting, bythe one or more computing devices, image data from an image data storagesystem and three-dimensional data for the point in time; and providing,by the one or more computing devices, the image data to the clientcomputing device.
 2. The method of claim 1, further comprising:generating a plurality of quadnodes, each quadnode being associated withthree-dimensional data; indexing the quadnodes based on location andpoint in time; and storing the indexed quadnodes in the image datastorage system, and wherein selecting the image data includesidentifying a quadnode based on the point in time.
 3. The method ofclaim 1, wherein selecting the image data includes three-dimensionaldata associated with objects that existed at the given one of the pointsin time proximate to the geographic location.
 4. The method of claim 1,wherein providing the image data includes: querying an index using thegiven one of the points in time to identify a storage location for theimage data; and retrieving the three-dimensional data from the storagelocation.
 5. The method of claim 1, further comprising: receivingthree-dimensional data for a geographic location, the receivedthree-dimensional data being associated with different points in time;and before providing the information identifying the set of availablepoints in time, determining the set of available points in time based onthe received three-dimensional data.
 6. The method of claim 1, whereinthe three-dimensional data for the point in time is three-dimensionalbuilding data.
 7. The method of claim 1, point in time is a range ofdates.
 8. A system comprising: memory storing image data andthree-dimensional data indexed by geographic location and time; and oneor more computing devices configured to: receive, from a clientcomputing device, a request for three-dimensional data, the requestidentifying a geographic location; provide information identifying a setof available points in time based on the identified geographic location;receive, from the client computing device, information identifying apoint in time in the set of available points in time; in response toreceiving the information identifying the point in time, select from thememory image data and three-dimensional data for the point in time; andprovide the image data to the client computing device.
 9. The system ofclaim 8, wherein the one or more computers are configured to: generate aplurality of quadnodes, each quadnode being associated withthree-dimensional data; index the quadnodes based on location and pointin time; and store the indexed quadnodes in the memory, and whereinselecting the image data includes identifying a quadnode based on thepoint in time.
 10. The system of claim 8, wherein the one or morecomputers are configured to select the image data by identifyingthree-dimensional data associated with objects that existed at the givenone of the points in time proximate to the geographic location.
 11. Thesystem of claim 8, wherein the one or more computers are configured to:querying an index using the given one of the points in time to identifya storage location for the image data; and retrieving thethree-dimensional data from the storage location.
 12. The system ofclaim 8, wherein the one or more computers are configured to: receivingthree-dimensional data for a geographic location, the receivedthree-dimensional data being associated with different points in time;and before providing the information identifying the set of availablepoints in time, determining the set of available points in time based onthe received three-dimensional data.
 13. The method of claim 8, whereinthe three-dimensional data for the point in time is three-dimensionalbuilding data.
 14. The system of claim 8, point in time is a range ofdates.
 15. A computer-readable storage device on which computer readableinstructions of a program are stored, the instructions, when executed bya processor, cause the processor to perform a method, the methodcomprising: receiving, from a client computer, a request forthree-dimensional data, the request identifying a geographic location;providing information identifying a set of available points in timebased on the identified geographic location; receiving, from the clientcomputing device, information identifying a point in time in the set ofavailable points in time; in response to receiving the informationidentifying the point in time, selecting image data from an image datastorage system and three-dimensional data for the point in time; andproviding the image data to the client computing device.
 16. The deviceof claim 15, wherein the method further comprises: generating aplurality of quadnodes, each quadnode being associated withthree-dimensional data; indexing the quadnodes based on location andpoint in time; and storing the indexed quadnodes in the image datastorage system, wherein selecting the image data includes identifying aquadnode based on the point in time.
 17. The device of claim 15, whereinselecting the image data includes identifying three-dimensional dataassociated with objects that existed at the given one of the points intime proximate to the geographic location.
 18. The device of claim 15,wherein providing the image data includes: querying an index using thegiven one of the points in time to identify a storage location for theimage data; and retrieving the three-dimensional data from the storagelocation.
 19. The device of claim 15, wherein the method furthercomprises: receiving three-dimensional data for a geographic location,the received three-dimensional data being associated with differentpoints in time; and before providing the information identifying the setof available points in time, determining the set of available points intime based on the received three-dimensional data.
 20. The device ofclaim 15, wherein the three-dimensional data for the point in time isthree-dimensional building data.